Depth image noise removal apparatus and method based on camera pose

ABSTRACT

Disclosed are a depth image noise removal apparatus based on a camera pose, which includes: a depth image obtaining unit for obtaining a plurality of depth images; a camera pose converting unit for converting camera poses of the plurality of depth images into a camera pose of a reference depth image; and a depth image filtering unit for filtering the reference depth image by using a weighted average of each pixel of the reference depth image, and a method using this apparatus.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to Korean Patent Application No.10-2012-0136849, filed on Nov. 29, 2012, and all the benefits accruingtherefrom under 35 U.S.C. §119, the contents of which in its entiretyare herein incorporated by reference.

BACKGROUND

1. Field

Embodiments of the present disclosure relate to noise removal of a depthimage, and more particularly, to a noise removal apparatus and method ofa depth image based on a camera pose.

2. Description of the Related Art

Recently, an image composing method using a depth image is widely usedfor reproducing a 3-dimensional image. Various kinds of cameras forphotographing such depth images have been developed and commercialized,but they have many problems such as a lot of noise. Generally, abilateral filter may be used for removing noise of a depth image. Thebilateral filter adds a range weight to a Gaussian blur to preserve edgeportions and blur the other regions. Therefore, as an improvement, atrilateral filtering method has been introduced. The trilateralfiltering method adjusts angle and width of a filtering window accordingto a gradient of a pixel.

However, the above filters adopt a filtering method employed in anexisting color image processing technique and are somewhat insufficientfor removing noise of a depth image.

SUMMARY

The present disclosure is directed to providing an apparatus and methodfor innovatively removing noise of a depth image by using a camera poseof the depth image.

In one aspect, there is provided a depth image noise removal apparatusbased on a camera pose, which includes: a depth image obtaining unit forobtaining a plurality of depth images; a camera pose converting unit forconverting camera poses of the plurality of depth images into a camerapose of a reference depth image; and a depth image filtering unit forfiltering the reference depth image by using a weighted average of eachpixel of the reference depth image.

The depth image filtering unit may determine the weighted average basedon a difference value of depth values and a distance value between eachpixel of the reference depth image and each pixel of another depthimage; and a difference value of camera poses for the plurality of depthimages.

The camera pose converting unit may perform obtaining relative camerapose values of the plurality of depth images based on the referencedepth image; and re-projecting the plurality of depth images to thereference depth image by calculating a re-projection matrix to therelative camera pose value.

The camera pose converting unit may obtain the relative camera posevalue by using an ICP algorithm.

The depth image filtering unit may calculate a difference value of thecamera pose by log-mapping the relative camera pose value.

The depth image filtering unit may perform successively generating afiltering window for a partial region of the reference depth image; andcalculating the weighted average from the filtering window.

In another aspect, there is provided a depth image noise removal methodbased on a camera pose, which includes: obtaining a plurality of depthimages; converting camera poses of the plurality of depth images into acamera pose of a reference depth image; and filtering the referencedepth image by using a weighted average of each pixel of the referencedepth image.

The weighted average may be determined based on a difference value ofdepth values and a distance value between each pixel of the referencedepth image and each pixel of another depth image; and a differencevalue of camera poses for the plurality of depth images.

Operation of converting camera poses of the plurality of depth imagesinto a camera pose of a reference depth image may include obtainingrelative camera pose values of the plurality of depth images based onthe reference depth image; and re-projecting the plurality of depthimages to the reference depth image by calculating a re-projectionmatrix to the relative camera pose value.

Operation of obtaining relative camera pose values may obtain therelative camera pose value by using an ICP algorithm.

The difference value of the camera pose may be calculated by log-mappingthe relative camera pose value.

Operation of filtering the reference depth image may includesuccessively generating a filtering window for a partial region of thereference depth image; and calculating the weighted average from thefiltering window.

According to the present disclosure, since depth images are successivelyphotographed and a relative camera pose is checked, several depth imagesmay be converted into the same viewpoint and blended, therebyeffectively removing a lot of noise.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects, features and advantages of the disclosedexemplary embodiments will be more apparent from the following detaileddescription taken in conjunction with the accompanying drawings inwhich:

FIG. 1 is a diagram showing a depth image noise removal apparatus basedon a camera pose according to an embodiment of the present disclosure;

FIG. 2 is a flowchart for illustrating a process for a camera poseconverting unit to convert a camera pose for a plurality of depth imagesaccording to an embodiment of the present disclosure;

FIG. 3 is a diagram for illustrating the concept of a plurality of depthimages where filtering windows are generated;

FIG. 4 is a diagram for illustrating a comparative pixel by which adepth image filtering unit calculates a distance and a depth valuebetween each pixel of a reference depth image and each pixel of anotherdepth image according to an embodiment of the present disclosure;

FIG. 5 is a flowchart for illustrating a depth image noise removalmethod based on a camera pose according to another embodiment of thepresent disclosure; and

FIG. 6 is a diagram comparatively showing a result where a PAT filteringmethod according to an embodiment of the present disclosure is appliedand a result where a bilateral filter is applied.

DETAILED DESCRIPTION

Hereinafter, embodiments of the present disclosure will be described indetail with reference to the accompanying drawings.

FIG. 1 is a diagram showing a depth image noise removal apparatus basedon a camera pose according to an embodiment of the present disclosure.Referring to FIG. 1, the depth image noise removal apparatus 100 basedon a camera pose according to this embodiment may be configured toinclude a depth image obtaining unit 101, a camera pose converting unit102, and a depth image filtering unit 103.

In an embodiment, the depth image obtaining unit 101 may include acamera capable of photographing a plurality of depth images and buildinga database. The depth image obtaining unit 101 may be a combination ofat least one color camera and depth camera or may also be composed ofonly depth cameras. In addition, the depth camera may employ a depthcamera using an infrared sensor (time of flight camera). However, in thepresent disclosure, the camera for obtaining a depth image is notlimited to the above and may include any device capable of obtaining adepth image.

In an embodiment, the depth image obtaining unit 101 may photograph aplurality of depth images, convert the images into electronic data, andtransfer the data to the camera pose converting unit 102.

The camera pose converting unit 102 may convert camera poses of aplurality of depth images transferred from the depth image obtainingunit 101 into a camera pose of a specific depth image. Here, thespecific depth image may be any one depth image selected from theplurality of depth images transferred from the depth image obtainingunit 101. In the specification, the specific depth image is called areference depth image. In other words, the camera pose converting unit102 may match camera poses of a plurality of depth images with a camerapose of a reference depth image. In addition, in the present disclosure,the reference depth image may be a frame subject to noise removal.

FIG. 2 is a flowchart for illustrating a process for a camera poseconverting unit to convert a camera pose for a plurality of depth imagesaccording to an embodiment of the present disclosure.

Referring to FIG. 2, the camera pose converting unit 102 may obtain acamera pose of each depth image from six depth images transferred fromthe depth image obtaining unit 101 (S22). Even though the presentdisclosure is illustrated based on six successive depth images as shownin FIG. 2, the present disclosure is not limited to the number of depthimages, and the number of depth images obtained may be different. Afterthat, the camera pose converting unit 102 may re-project a camera poseof another depth images to the reference depth image (S23).

In other words, when obtaining a camera pose of each depth image, thecamera pose converting unit 102 obtains relative camera pose values fora plurality of depth images based on the reference depth image. Afterthat, the camera pose converting unit 102 calculates a re-projectionmatrix to the relative camera pose value so that the plurality of depthimages may be re-projected according to the reference depth image. Inaddition, the camera pose converting unit 102 may obtain the relativecamera pose value by using a point-to-plane Iterative Closest Point(ICP) algorithm.

In an embodiment of the present disclosure, the depth image filteringunit 103 may filter the reference depth image by using a weightedaverage of each pixel of the reference depth image. Here, the depthimage filtering unit 103 may successively generate filtering windowswith respect to the reference depth image subject to noise removal, andcalculate an weighted average among the filtering windows. The filteringwindows may be generated for one portion of the reference depth imageand then successively generated for all regions of the reference depthimage so that the entire reference depth image may be filtered. However,the filtering window may also be generated for a plurality of depthimages other than the reference depth image.

FIG. 3 is a diagram for illustrating the concept of a plurality of depthimages where filtering windows are generated. Referring to FIG. 3, aplurality of depth images where camera poses are aligned is depicted. Afiltering window is generated for a portion of the depth image, and thedepth image filtering unit 103 may filter the filtering window. Inaddition, the depth image filtering unit 103 may successively generatefiltering windows such as a first filtering window 311, a secondfiltering window 312 and a third filtering window 313 and successivelyperform local filtering thereto with respect to the reference depthimage 310. In this filtering process, information of pixels of otherdepth images (for example, depth images 320, 330, 340) may be used.

The depth image filtering unit 103 according to an embodiment of thepresent disclosure may determine a weighted average based on (a) adifference value of depth values between each pixel of the referencedepth image and each pixel of another depth image, (b) a distance valuebetween each pixel of the reference depth image and each pixel ofanother depth image, and (c) a difference value of camera poses for theplurality of depth images.

In other words, the depth image filtering unit 103 may obtain a filteredvalue with respect to a pixel (p) of the reference depth image (k)according to Equation 1 below.

$\begin{matrix}{{f( I_{p}^{k} )} = {\frac{1}{w_{p}^{k}}{\sum\limits_{i \in C}{\sum\limits_{q \in S}{{G_{\sigma_{s}}( {{p - q}} )}{G_{\sigma_{r}}( {{I_{p}^{k} - I_{q}^{i}}} )}{G_{\sigma_{c}}( {{\log( T_{ki} )}} )}I_{q}^{i}}}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

Here, k represents a reference depth image, and I represents depthimages other than the reference depth image. In addition, w_(p) ^(k)represents a sum of all weights in the corresponding window, and I_(q)^(i) represents a depth value of a pixel q of the depth image i. Inaddition, details of G_(σ) _(s) (∥p−q∥), G_(σ) _(r) (|I_(p) ^(k)−I_(q)^(i)|), G_(σ) _(c) (∥log (T_(ki))∥) are represented in detail inEquations 2 to 4 below.

$\begin{matrix}{{G_{\sigma_{s}}( {{p - q}} )} = {\mathbb{e}}^{{- \frac{1}{2}}{(\frac{{p - q}}{\sigma_{s}})}^{2}}} & {{Equation}\mspace{14mu} 2} \\{{G_{\sigma_{r}}( {{I_{p}^{k} - I_{q}^{i}}} )} = {\mathbb{e}}^{{- \frac{1}{2}}{(\frac{{I_{p}^{k} - I_{q}^{i}}}{\sigma_{r}})}^{2}}} & {{Equation}\mspace{14mu} 3} \\{{G_{\sigma_{c}}( {{\log( T_{ki} )}} )} = {\mathbb{e}}^{{- \frac{1}{2}}{(\frac{{\log{(T_{ki})}}}{\sigma_{c}})}^{2}}} & {{Equation}\mspace{14mu} 4}\end{matrix}$

Here, T_(ki) represents a relative camera pose value between two depthimages. Here, in order to obtain the size of T_(ki), a norm obtained bylog-mapping may be calculated.

FIG. 4 is a diagram for illustrating a comparative pixel by which adepth image filtering unit calculates a distance and a depth valuebetween each pixel of a reference depth image and each pixel of anotherdepth image according to an embodiment of the present disclosure.Referring to FIG. 4, a filtering window 311 for the reference depthimage and a filtering window 321 for another successive depth image aredepicted. A filtering process for a pixel 41 of a filtering referencedepth image will be described. In detail, the depth image filtering unit103 performs a filtering process described later with respect to eachpixel in the filtering window. In addition, since the filtering windowis generated for the entire area of the reference depth image asdescribed above, it will be understood that the following filteringprocess is performed to all pixels of the reference depth image.

In an embodiment, the depth image filtering unit 103 may calculate adifference value (a) of depth values between each pixel of the referencedepth image and each pixel of another depth image. For example, thedepth image filtering unit may determine the difference value bycomparing the depth value of the pixel 41 with depth values of allpixels in the filtering window 321 of another depth image. FIG. 4exemplarily shows a pixel 43 and a pixel 44 in the filtering window 321of another depth image.

In addition, the depth image filtering unit 103 may calculate a distancevalue (b) between each pixel of the reference depth image and each pixelof another depth image. In case of calculating the distance value (b)with respect to the pixel 41, a distance value may be calculated basedon a location of another depth image, matched with the correspondingpixel 41 of the reference depth image. For example, the distance valuefrom the pixel 41 to the pixel 43 may be calculated as a distance valuefrom the pixel 42 to the pixel 43 as shown in FIG. 4. Similarly, thedistance value from the pixel 41 to the pixel 44 may be calculated as adistance value from the pixel 42 to the pixel 44. Since the camera posechanging unit 102 aligns camera poses of the plurality of depth imagesinto a single camera pose and the filtering windows are arrangedregularly in the aligned camera pose as shown in FIG. 3, it may beunderstood that the pixel 41 and the pixel 42 are located at relativelyidentical coordinates with respect to each filtering window.

In an embodiment, the depth image filtering unit 103 may determine adifference value (c) of a camera pose with respect to the plurality ofdepth images based on the reference depth image. In detail, the depthimage filtering unit 103 may determine a difference value of the camerapose by log-mapping the relative camera pose value as shown in Equation4. In addition, the depth image filtering unit 103 may obtain therelative camera pose value by using an ICP algorithm. As a result, itmay be understood that a depth image obtained at a pose more similar tothe reference depth image has a greater weight.

FIG. 5 is a flowchart for illustrating a depth image noise removalmethod based on a camera pose according to another embodiment of thepresent disclosure. The depth image noise removal method based on acamera pose according to this embodiment includes obtaining a pluralityof depth images (S1), converting camera poses of the plurality of depthimages into a camera pose of the reference depth image (S2), andfiltering the reference depth image by using a weighted average of eachpixel of the reference depth image (S3).

In an embodiment, Operation of obtaining a plurality of depth images(S1) may be performed with the same function as the depth imageobtaining unit 101 described above or may be performed by the depthimage obtaining unit 101. In addition, Operation of converting cameraposes of the plurality of depth images into a camera pose of thereference depth image (S2) may include obtaining relative camera posevalues of the plurality of depth images based on the reference depthimage, and re-projecting the plurality of depth images to the referencedepth image by calculating a re-projection matrix to the relative camerapose value.

In an embodiment, a method for determining a weighted average isidentical to the above, and Operation of filtering the reference depthimage (S3) may include successively generating a filtering window for apartial region of the reference depth image, and calculating theweighted average from the filtering window.

In other words, the noise removal method according to an embodiment ofthe present disclosure obtains n number of depth images including thereference depth image and its back and forth before, which are subjectto noise removal, and calculates relative camera locations from thereference depth image, with respect to n−1 number of back and forthdepth images, excluding the reference depth image. In addition, n−1number of back and forth depth images may be re-projected to a cameralocation of the reference depth image, and n number of depth imagesconverted as if being obtained at a single place are blended by means ofthe filtering method described above. The filtering method may also becalled a Pose Aware Trilateral (PAT) filtering method.

FIG. 6 is a diagram comparatively showing a result where a PAT filteringmethod according to an embodiment of the present disclosure is appliedand a result where a bilateral filter is applied.

Referring to FIG. 6, it may be understood that more noise is removed inFIG. 6( c) depicting the resultant product of the PAT filteringaccording to an embodiment of the present disclosure, compared with FIG.6( b) depicting the resultant product of a bilateral filter wherein FIG.6( a) is diagram showing a image before the filtering.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the presentdisclosure. As used herein, the singular forms “a”, “an” and “the” areintended to include the plural forms as well, unless the context clearlyindicates otherwise. It will be further understood that the terms“comprises” and/or “comprising”, or “includes” and/or “including” whenused in this specification, specify the presence of stated features,regions, integers, steps, operations, elements, and/or components, butdo not preclude the presence or addition of one or more other features,regions, integers, steps, operations, elements, components, and/orgroups thereof.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art. It will be further understood that terms,such as those defined in commonly used dictionaries, should beinterpreted as having a meaning that is consistent with their meaning inthe context of the relevant art and the present disclosure, and will notbe interpreted in an idealized or overly formal sense unless expresslyso defined herein. In the drawings, like reference numerals denote likeelements.

Exemplary embodiments now will be described more fully hereinafter withreference to the accompanying drawings, in which exemplary embodimentsare shown. The present disclosure may, however, be embodied in manydifferent forms and should not be construed as limited to the exemplaryembodiments set forth therein. Rather, these exemplary embodiments areprovided so that the present disclosure will be thorough and complete,and will fully convey the scope of the present disclosure to thoseskilled in the art. In the description, details of well-known featuresand techniques may be omitted to avoid unnecessarily obscuring thepresented embodiments.

However, in the description, details of well-known features andtechniques may be omitted to avoid unnecessarily obscuring the presentedembodiments. In addition, the shape, size and regions, and the like, ofthe drawing may be exaggerated for clarity and may not mean the actualdimension.

The embodiments described in the specification may be implemented ashardware entirely, hardware partially and software partially, orsoftware entirely. In the specification, the term “unit”, “module”,“device”, “system” or the like indicates a computer-related entity likehardware, a combination of hardware and software, or software. Forexample, the term “unit”, “module”, “device”, “system” or the like usedin the specification may be a process, a processor, an object, anexecutable file, a thread of execution, a program, and/or a computer,without being limited thereto. For example, both a computer and anapplication executed in the computer may correspond to the term “unit”,“module”, “device”, “system” or the like in the specification.

The embodiments have been described with reference to the flowchartshown in the figure. For brief explanation, the method has beenillustrated and described as a series of blocks, but the presentdisclosure is not limited to the order of the blocks. In other words,some blocks may be executed simultaneously with other blocks or in adifferent order from those illustrated and described in thisspecification, and various diverges, flow paths, block sequences mayalso be implemented if they give the equivalent or similar results. Inaddition, in order to implement the method described in thespecification, it is also possible not to demand all blocks. Further,the method for predicting a surgery stage may be implemented in the formof a computer program for executing a series of processes, and thecomputer program may also be recorded on a computer-readable recordingmedium.

Though the present disclosure has been described with reference to theembodiments depicted in the drawings, it is just an example, and itshould be understood by those skilled in the art that variousmodifications and equivalents can be made from the disclosure. However,such modifications should be regarded as being within the scope of thepresent disclosure. Therefore, the true scope of the present disclosureshould be defined by the appended claims.

What is claimed is:
 1. A depth image noise removal apparatus based on acamera pose, comprising a computer storing and executing a computerreadable program, the computer readable program comprising: a depthimage obtaining unit configured to obtain depth images; a camera poseconverting unit configured to convert camera poses of the depth imagesinto a camera pose of a reference depth image; and a depth imagefiltering unit configured to filter the reference depth image by using aweighted average of each pixel of the reference depth image.
 2. Thedepth image noise removal apparatus based on a camera pose according toclaim 1, wherein the depth image filtering unit determines the weightedaverage based on a difference value of depth values and a distance valuebetween each pixel of the reference depth image and each pixel ofanother depth image; and a difference value of camera poses for thedepth images.
 3. The depth image noise removal apparatus based on acamera pose according to claim 2, wherein the camera pose convertingunit performs obtaining relative camera pose values of the depth imagesbased on the reference depth image; and re-projecting the depth imagesto the reference depth image by calculating a re-projection matrix tothe relative camera pose value.
 4. The depth image noise removalapparatus based on a camera pose according to claim 3, wherein thecamera pose converting unit obtains the relative camera pose value byusing an ICP algorithm.
 5. The depth image noise removal apparatus basedon a camera pose according to claim 3, wherein the depth image filteringunit calculates a difference value of the camera pose by log-mapping therelative camera pose value.
 6. The depth image noise removal apparatusbased on a camera pose according to claim 2, wherein the depth imagefiltering unit performs successively generating a filtering window for apartial region of the reference depth image; and calculating theweighted average from the filtering window.
 7. The depth image noiseremoval apparatus based on a camera pose according to claim 1, whereinthe computer includes a camera capable of photographing depth images andbuilding a database.
 8. The depth image noise removal apparatus based ona camera pose according to claim 1, wherein the computer includes acombination of a color camera and depth camera.
 9. The depth image noiseremoval apparatus based on a camera pose according to claim 8, whereinthe depth camera has an infrared sensor.
 10. The depth image noiseremoval apparatus based on a camera pose according to claim 1, whereinthe computer comprises depth cameras.
 11. The depth image noise removalapparatus based on a camera pose according to claim 10, wherein a depthcamera has an infrared sensor.
 12. A depth image noise removal methodbased on a camera pose, comprising: obtaining depth images; convertingcamera poses of the depth images into a camera pose of a reference depthimage; and filtering the reference depth image by using a weightedaverage of each pixel of the reference depth image.
 13. The depth imagenoise removal method based on a camera pose according to claim 12,wherein the weighted average is determined based on a difference valueof depth values and a distance value between each pixel of the referencedepth image and each pixel of another depth image; and a differencevalue of camera poses for the depth images.
 14. The depth image noiseremoval method based on a camera pose according to claim 13, whereinsaid converting of camera poses of the depth images into a camera poseof a reference depth image includes obtaining relative camera posevalues of the depth images based on the reference depth image; andre-projecting the depth images to the reference depth image bycalculating a re-projection matrix to the relative camera pose value.15. The depth image noise removal method based on a camera poseaccording to claim 14, wherein said obtaining of relative camera posevalues obtains the relative camera pose value by using an ICP algorithm.16. The depth image noise removal method based on a camera poseaccording to claim 14, wherein the difference value of the camera poseis calculated by log-mapping the relative camera pose value.
 17. Thedepth image noise removal method based on a camera pose according toclaim 13, wherein said filtering of the reference depth image includessuccessively generating a filtering window for a partial region of thereference depth image; and calculating the weighted average from thefiltering window.